Systems and methods for railway asset management

ABSTRACT

Systems and methods for railway asset management. The methods comprise: using a virtual reality device to recognize and collect real world information about railway assets located in a railyard; and using the real world information to (i) associate a railway asset to a data collection unit, (ii) provide an individual with an augmented reality experience associated with the railyard and/or (iii) facilitate automated railyard management tasks.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application Ser. No. 63/048,871 which was filed on Jul. 7, 2020and U.S. Provisional Patent Application Ser. No. 63/153,652 which wasfiled on Feb. 25, 2021. The contents of these Provisional PatentApplications are incorporated herein by reference in their entirety.

BACKGROUND Statement of the Technical Field

The present document generally relates to railway asset managementsystems. More particularly, the present solution relates to implementingsystems and methods for (i) associating a railway asset to a datacollection unit and/or (ii) Augmented Reality (AR) based railyardmanagement.

Description of the Related Art

In railcar transport systems, railcars are used to carry loose bulkcommodities, liquid commodities and/or other types of goods by rail.Such goods may be loaded and unloaded at railyards. The locations of therailcars may change during different phases of a railyard managementprocess. The phases include an inbound phase, a load/unload phase, andan outbound phase. A railyard map and scheduling system are used tocoordinate movements of the railcars through the multiple tracks/pathsof the railyard. The railyard map shows the locations of the railcars inthe railyard, and any changes in the same as the railcars move throughthe railyard. The railyard map is updated manually using informationobtained by individuals who are present in the railyard and who inspectthe railcars. This manual process is time consuming, error prone anddangerous to personnel carrying out the various inspection processes.

Additionally, various data collection units are typically coupled torailcars. The data collection units are communicatively coupled to eachother via the Internet, and therefore are collectively referred to as anInternet of Things (IoT). Companies that sell IoT based products forrailcar transport systems need to associate the data collection unit(s)in the railcar transport systems with the reporting marks of therailcars to which they are coupled. These companies require theircustomers to manually (i) install the data collection unit(s) on therailcar, (ii) document the serial number(s) of the installed datacollection unit(s), and (iii) document the reporting mark(s) on therailcar(s) on which the data collection unit(s) was(were) installed.Operations (i), (ii) and (iii) are often performed by more than oneperson. This is an extremely manual and error prone process that doesnot allow individuals to view such installation information inreal-time.

SUMMARY

The present disclosure relates to implementing systems and methods forrailway asset management. The methods comprise: capturing an image ofthe railway asset using a Mobile Communication Device (MCD); convertingthe image into an electronic editable image of a mark on the railwayasset; wirelessly communicating the electronic editable image from theMCD to a data collection unit which is installed on the railway asset;communicating first information from the data collection unit to aremote computing device via a first network communication (the firstinformation comprises at least the electronic editable image); comparingthe first information to second information to determine whether a matchexists therebetween by a given amount; and validating that the datacollection unit was installed on the railway asset when a match isdetermined to exist between the first and second information by thegiven amount. The second information may be communicated from the MCD tothe remote computing device with a second network communication. Thesecond information comprises the image, pre-stored information retrievedfrom a datastore of a railway asset management system, and/or adatastore of another system.

The methods may also comprise: providing an electronic notification to auser of a computing device that the install was completed successfullywhen a match is determined to exist between the first and secondinformation by the given amount; providing an electronic notification toa user of a computing device that the install was not completedsuccessfully when a match is determined to not exist between the firstand second information by the given amount; storing the firstinformation in a datastore responsive to a validation that the datacollection unit was installed on the railway asset; associating a uniqueidentifier of the data collection unit with the mark in a datastoreresponsive to a validation that the data collection unit was installedon the railway asset; and/or discarding the first information when adetermination is made that the first and second information do not matcheach other by the given amount.

The methods may further comprise: performing monitoring operations bythe data collection unit in response to the validating to monitor atleast one of an operational performance of the railway asset, a statusof at least one component of the railway asset, an amount of loaddisposed in or on the railway asset, and a condition of an environmentsurrounding the railway asset; and/or analyzing information from themonitoring operations to determine whether at least one of theoperational performance of the railway asset, the status of the at leastone component of the railway asset, and the condition of the environmentsurrounding the railway asset is acceptable. The operational performanceof the railway asset, the status of the at least one component of therailway asset, or the condition of the environment surrounding therailway asset may be considered acceptable when at least one of a hatch,a valve, a door, wheels, brakes, axles, a railcar connection isoperating in an expected manner, an amount of load disposed in or on therailway asset is within a given range, and no leaks have been detectedbased on odors, scents or smells detected by a sensor of the datacollection unit. One or more of the following operations may beperformed when a determination is made that at least one of theoperational performance of the railway asset, the status of the at leastone component of the railway asset, and the condition of the environmentsurrounding the railway asset is unacceptable: remove the railway assetfrom use temporarily; order a component based on an analysis of theimage; schedule maintenance for the railway asset; and adjust an amountof load in or on the railway asset. Transportation activities for therailway asset may be scheduled when a determination is made that atleast one of the operational performance of the railway asset, thestatus of the at least one component of the railway asset, and thecondition of the environment surrounding the railway asset isacceptable.

The present document also concerns methods for AR based railyardmanagement. The methods comprise: using a virtual reality device torecognize and collect real world information about railway assetslocated in a railyard; and using the real world information to providean individual with an augmented reality experience associated with therailyard and facilitate automated railyard management tasks. Theautomated railyard management tasks can include, but are not limited to,validating a train consist, validating information disposed on therailway assets, detecting locations of the railway assets in therailyard, updating a map of the railyard, monitoring states of therailway assets while in the railyards, detecting damage to the railwayassets in the railyards, detecting hazards of the railway assets,predicting future issues with the railway assets based on machinelearned information, performing maintenance checks for components of therailway assets, scheduling maintenance for the railway assets,facilitating maintenance of railway assets using a robotic manipulatorwhich is remotely controlled via a virtual reality environment,performing security checks for the railyard, performing security checksfor the railway assets, performing compliance checks for the railwayassets, and/or providing notifications to individuals.

The augmented reality experience is provided to the individual by:allowing a real world environment of the railyard to be visible to anindividual who is wearing the virtual reality device; generatingholographic image data using the real world information; and overlayingthe holographic image data on the visible real world environment. Thereal world information can include, but is not limited to, locations ofthe railway assets, information disposed on the railway assets (e.g.,railcar marks), physical conditions of the railway assets (e.g., havingor absent of dents, cracks, wear, etc. and/or having operative ordefective brakes, hatches, discharge gates, ports, etc.), operatingstates of components of the railway assets (e.g., open, closed, sealed,latched, unlatched, etc.), and/or physical conditions of the componentsof the railway assets (e.g., having or absent of dents, cracks, wear,tears, etc. and/or having worn brakes, etc.).

The methods may also comprise: performing a machine learning algorithmusing the real world information to predict a future event or conditionrelating to at least one railway asset of the railway assets (e.g.,predicted mechanical fault and/or derailment); causing an action (e.g.,schedule maintenance, order part, temporarily remove from use, etc.) tobe taken in relation to the railway asset based on the predicted futureevent or condition; using the real world information to facilitate aninspection of the railway assets by an individual remote from therailyard; and/or using real world information to facilitate a remotecontrol of a robotic manipulator located in the railyard.

The implementing systems can comprise: a processor; and a non-transitorycomputer-readable storage medium comprising programming instructionsthat are configured to cause the processor to implement a method forrailway asset management.

BRIEF DESCRIPTION OF THE DRAWINGS

The present solution will be described with reference to the followingdrawing figures, in which like numerals represent like items throughoutthe figures.

FIG. 1 provides an illustration of an illustrative system.

FIG. 2 provides an illustration of an illustrative MCD.

FIG. 3 provides an illustration of an illustrative computing device.

FIGS. 4A-4B (collectively referred to herein as “FIG. 4”) provide a flowdiagram of an illustrative method for associating a railcar to a datacollection unit.

FIG. 5 provides an illustration of an illustrative system implementingthe present solution.

FIG. 6 provides an illustration of an illustrative track arrangement ofa railyard.

FIGS. 7 and 8 provide illustrations that are useful for understandinginformation, marking and decals of a railcar.

FIG. 9 provides an illustration of an illustrative railyard map.

FIGS. 10-12 each provides an illustration of an illustrative GraphicalUser Interface (GUI).

FIG. 13 provides an illustration that is useful for understandingautomated maintenance inspections of the present solution.

FIG. 14 provides an illustration showing railcars on a plurality oftracks in a railyard.

FIGS. 15-18 each provides an illustration of an illustrative GUI.

FIG. 19 provides a flow diagram of an illustrative method for railyardmanagement.

DETAILED DESCRIPTION

The present invention is described with reference to the attachedfigures. The figures are not drawn to scale and they are provided merelyto illustrate the instant invention. Several aspects of the inventionare described below with reference to example applications forillustration. It should be understood that numerous specific details,relationships, and methods are set forth to provide a full understandingof the invention. One having ordinary skill in the relevant art,however, will readily recognize that the invention can be practicedwithout one or more of the specific details or with other methods. Inother instances, well-known structures or operation are not shown indetail to avoid obscuring the invention. The present invention is notlimited by the illustrated ordering of acts or events, as some acts mayoccur in different orders and/or concurrently with other acts or events.Furthermore, not all illustrated acts or events are required toimplement a methodology in accordance with the present invention.

As noted above, companies that sell IoT based products for railcartransport systems need to associate the data collection unit(s) in therailcar transport systems with the reporting marks of the railcars towhich they are coupled. These companies require their customers tomanually (i) install the data collection unit(s) on the railcar, (ii)document the serial number(s) of the installed data collection unit(s),and (iii) document the reporting mark(s) on the railcar(s) on which thedata collection unit(s) was(were) installed. This is an extremely manualand error prone process that does not allow individuals to view suchinstallation information in real-time.

The present solution solves these drawbacks of the conventionalsolutions. The present solution generally provides implementing systemsand methods for automatically associating a reporting mark of a railcarto data collection unit(s) in the field as an individual is installingthe data collection unit(s) on the railcar. This automatic associationprocess is achieved using an MCD in the possession of the individual.The MCD can include, but is not limited to, a smart phone, a PersonalDigital Assistant (PDA), a personal computer, a laptop, a tablet, smartglasses, a virtual reality device, and/or a data collection unit. Themanner in which the MCD facilitates the automatic association processwill become evident as the discussion progresses. This associationbetween railcar reporting marks and data collection units allows endusers to easily and quickly access field data associated with the same.This field data can include, but is not limited to, tare weight(s),maximum weight(s), certification data (e.g., certification referencenumber(s) and/or certification date(s)), data collected from sensors(e.g., a wired sensor or WSN) installed on the railcar (e.g., commoditytemperature sensor data, hatch status data, bearing temperature data,and/or load status data), and/or railcar/locomotive component data(e.g., make, model, serial number, wheel size, etc.).

The present solution also provides an AR based solution to automaterailyard management tasks. The automated railyard management tasksinclude, but are not limited to, validating train consists, validatinginformation and/or markings on railway assets, detecting/trackinglocations of railway assets in railyards, updating railyard maps,monitoring states of the railway assets while in the railyards (e.g.,inbound state, loading state, unloading state, maintenance state,fueling state, cleaning state, and/or outbound state), detecting damageto railway assets in the railyards, detecting hazards to railway assets,hazards on railway assets (e.g., a tripping or fall hazard caused by abroken or missing rung on a ladder), predicting future issues withrailway assets based on machine learned information (e.g., a predictedderailment of a railcar based on detected state(s) of components thereof(e.g., a detected crack or other mechanical fault in a wheel, axle,bearing, etc.), or predicted component failure (e.g., based on lifeexpectancy thereof, component type, duration of use, and/or amount ofwear/tear from use), performing maintenance checks for variouscomponents of the railway assets (e.g., wheels, appliances, ladders,etc.), scheduling maintenance for the railway assets, facilitatingmaintenance of railway assets using robotic manipulator(s) (e.g.,articulating arms) which are remotely controlled via a Virtual Reality(VR) environment, performing security checks, performing compliancechecks (e.g., regulatory, shipping, customer, etc.), and providingalerts/notifications to relevant parties/individuals.

Accordingly, the methods of the present solution generally involve:using an AR device to recognize and collect information about thelocations and states of railway assets in a railyard; and using thecollected information to facilitate automated railyard management tasks(e.g., such as those listed above). Railway assets may include, but arenot limited to, railcars, locomotives, rail maintenance equipment,containers, and International Standards Organization (ISO) tanks. Inthis document, a railcar will be used for illustrative purposes. Arailcar can include, but is not limited to, a hopper car or tank car.

The present solution can be used in various applications. Suchapplications include, but are not limited to, installation trainingapplications, installer applications, railcar management applications,railcar maintenance applications, railcar certification applications,railcar transport applications, and/or any other application in whichlocations and/or operational states of assets need to be monitoredand/or tracked. For example, the present solution can be employed in thesystems described in, for example, U.S. Pat. No. 10,850,755 to Lefebvreet al. (“the '755 patent”) which issued on Dec. 1, 2020, U.S. Pat. No.10,710,619 which issued on Jul. 14, 2020, U.S. Pat. No. 9,663,092 whichissued on May 30, 2017, U.S. Pat. No. 10,137,915 which issued on Nov.27, 2018, and U.S. Pat. No. 9,981,673 which issued May 29, 2018. Thecontent of the listed Patents are incorporated herein in their entirety.

Illustrative Systems for Associating a Railcar to a Data Collection Unit

Referring now to FIG. 1, there is provided an illustration of anillustrative system 100 implementing the present solution. System 100comprises a railcar 102 coupled to a locomotive 120. An individual 104is provided to install at least one data collection unit 118 on therailcar 102. The data collection unit 118 can include, but is notlimited to, a gateway, a sensor (e.g., wired sensor or a wirelesssensor), and/or a Communication Management Unit (CMU). The sensor caninclude, but is not limited to, a temperature sensor (e.g., ambientand/or wheel bearing), a weight sensor (e.g., for measuring a weight ofcommodities loaded on the railcar), a force sensor (e.g., for measuringforces experienced by the railcar during coupling), a location sensor(e.g., for specifying a location of the railcar in a rail yard and/or ona train track), a humidity sensor, an odor/scent/smell sensor (e.g., fordetecting leaks or spills of hazardous chemicals), a light sensor, anair pressure sensor, a vibration sensor, an accelerometer, a travelingspeed sensor, a traveling direction sensor, a hatch position sensor, abrake pressure sensor, a hand brake on/off sensor, a railcar load sensor(e.g., to indicate whether the railcar is full or empty), a commoditysensor (e.g., to indicate a presence, state and/or type of commoditiesloaded on the railcar), a bearing fault sensor, a piezo-electric sensor,an acoustic sensor, and/or a track damage detection sensor. The couplingcan be achieved via a weld, a mechanical coupler (e.g., a screw, a bolt,a nut, a clamp, a latch, bracket, etc.), and/or an adhesive.

The individual 104 may have an MCD 106 in his(her) possession. The MCD106 can include, but is not limited to, a mobile phone, a smart phone, apersonal computer, a laptop, a tablet, a PDA, a smart watch, smartglasses, a smart helmet, and/or a smart visor (e.g., coupled to a hatand/or a vehicle such as a personal transporter). During installation ofthe data collection unit 118, the individual 104 uses the MCD 106 tomanually input a mark disposed on the railcar 102 and/or capture animage of the mark 130 disposed on the railcar 102. The mark 130 caninclude one or more letters, numbers and/or symbols.

In some scenarios, the captured image is processed by the MCD 106 to atleast (i) detect the mark within the captured image and (ii) generate anelectronic and editable image of a mark (e.g., a railcar mark) on arailway asset (e.g., a railcar) based on the detected mark (e.g., astring of letters, numbers and/or symbols) within the captured image.Operation (i) can be achieved using any known or to be known OpticalCharacter Recognition (OCR) algorithm. The OCR algorithm may also beused to obtain other railcar information about the railcar 102 from thecaptured image. This other railcar information can include, but is notlimited to, tare weight(s), maximum weight(s), certification referencenumber(s), certification date(s), data collected from sensors installedon the railcar (e.g., commodity temperature sensor data, hatch statusdata, bearing temperature data, and/or load status data),railcar/locomotive component data (e.g., make, model, serial number,wheel size, etc.), and/or maintenance information (e.g., date/time oflast maintenance and/or type of maintenance performed).

The individual 104 compares the electronic editable mark to the mark 130disposed on the railcar 102. If a match does not exit, then theindividual 104 modifies the electronic editable version of the mark sothat the same accurately and/or completely represents the actual mark130. Techniques for modifying/editing images and/or strings ofletters/numbers/symbols are well known in the art. Any known or to beknown technique for modifying/editing images and/or strings ofletters/numbers/symbols can be used herein without limitation. Forexample, the user can perform user-software interactions via a touchscreen, a keypad and/or other input means for modifying contentpresented in a display. Railcar mark information specifying the mark isthen sent from the MCD 106 to the data collection unit 118 and/orwireless sensor node(s) 114 via a Near Field Communication (NFC) and/ora Short Range Communication (SRC) 140, 141. The other railcarinformation may also be sent along with the railcar mark information.

The data collection unit 118 then sends the railcar mark information,the other railcar information and/or metadata to a remote server 110 viaa network 108 (e.g., the Internet, a cellular network, a radio network,a satellite based network) (as shown by communication links 152, 142 and146), a wireless sensor node 114 of railcar 102 and/or a gateway 122 ofthe locomotive 120 (as shown by communication links 142, 148, 150, 152).The metadata can include, but is not limited to, a unique identifier ofthe data collection unit 118, a unique identifier of the individual 104and/or MCD 106, time information indicating when the data collectionunit 118 was installed on the railcar 102, time information indicating atime when the railcar mark information was received at and/ortransmitted from the data collection unit 118, and/or locationinformation indicating a location of the data collection unit 118 at thetime of receipt and/or transmission of the railcar mark information.

At the remote server 110, the railcar mark information and/or metadataare stored in a datastore 112 and/or presented to a user thereof. In thedatastore 112, the unique identifier of the data collection unit 118 isassociated with the mark 130 of the railcar 102 and/or other informationassociated with the railcar 102 (e.g., sensor data, weight(s),certification information and/or maintenance information).

Additionally or alternatively, an electronic message is sent to one ormore computing devices 116 located at a site at which the railcar 102resides and/or at a site that is remote from the site at which therailcar 102 resides. The electronic message can include, but is notlimited to, an email message, a website alert, an Internet instantmessage, and/or a text message from the server 110. The electronicmessage provides a notification that the data collection unit 118 hasbeen properly installed on the railcar 102 by the individual 104. Theelectronic message can be sent from the server 110 and/or MCD 106 to thecomputing device 116 as shown by communication links 146, 154 and/or144, 154.

The MCD 106 may additionally or alternatively send the captured image ofthe railcar 102, the railcar mark information, the other railcarinformation and/or metadata to the remote server 110 via the network108, as shown by communication links 144 and 146. The metadata caninclude, but is not limited to, a unique identifier of the MCD 106, aunique identifier of the individual 104, a unique identifier of the datacollection unit 118, time information specifying a time at which thedata collection unit 118 was installed, time information specifying atime at which the captured image was transmitted from the MCD 106 todata collection unit 118 and/or server 110, and/or location informationspecifying a location of the MCD 106 at the time of installation of thedata collection unit 118.

At the server 110, the information received from the MCD 106 may becompared to the information received from the data collection unit 118.If the information received from the MCD 106 matches the informationreceived from the data collection unit 118 by a certain amount (e.g.,50-100%), then the server validates that the data collection unit 118was installed on the railcar 102. When this validation is made, therailcar mark information and/or metadata (from the data collection unit118 and/or MCD 106) are stored in a datastore 112 and/or presented to auser of the computing device(s) 110, 116. In the datastore 112, theunique identifier of the data collection unit 118 is associated with themark 130 of the railcar 102 and/or other information associated with therailcar 102 (e.g., sensor data, weight(s), certification informationand/or maintenance information). Additionally or alternatively, theelectronic message is sent to the computing device(s) 116 from theserver 110 and/or the MCD 106.

The above described operations of system 100 help to automate theprocess of associating the railcar 102 and a data collection unit 118 inthe field (for example, a railyard), and minimizes typographical errorsmade by individuals manually entering the complete railcar mark into thesystem. Additionally, the individual 104 is able to relatively quicklyretrieve railcar data (e.g., tare weights, maximum weight, certificationdata and/or data collected by sensor(s)) and to correlate the same withother data (e.g., installation data). The present solution also solvesthe issue of associating the railcar 102 and data collection unit 114when a network 108 is not available. The manner in which this issue isaddressed will become evident as the discussion progresses.

Referring now to FIG. 2, there is provided an illustration of anillustrative architecture for a communication device 200. The MCD 106,wireless sensor node 114, data collection unit 118, and/or gateway 122of FIG. 1 is/are the same as or similar to the communication device 200of FIG. 2. As such, the discussion of communication device 200 issufficient for understanding devices 106, 114, 118, 122 of FIG. 1.

Communication device 200 may include more or less components than thoseshown in FIG. 2. However, the components shown are sufficient todisclose an illustrative hardware architecture implementing the presentsolution. The hardware architecture of FIG. 2 represents one embodimentof a representative communication device configured to facilitate anassociation of a railcar to a data collection unit. The operations andfunctions can include, but are not limited to, communicating informationto/from external devices, perform OCR based processes to detect objectsin captured images, process information extracted from images, andoutput information to a user of the communication device 200.

As shown in FIG. 2, the communication device 200 comprises an antenna202 for receiving and transmitting wireless signals (e.g., RF signals,cellular signals, and/or satellite signals). A transceiver switch 204selectively couples the antenna 202 to a transmit circuit 206 and areceive circuit 208 in a manner familiar to those skilled in the art.The present solution is not limited in this regard. The communicationdevice 200 can alternatively comprise one or more antennas for eachtransceiver circuit 206 and 208, and therefore may be absent of thetransceiver switch 204 for selectively connecting the transmit circuitand receive circuit to a common antenna.

Transmit and receive circuits are well known in the art, and thereforewill not be described in detail herein. Still, it should be understoodthat the transmit circuit 206 is configured to (i) cause information tobe transmitted to an external device (e.g., server 110 of FIG. 1) viawireless signals and (ii) process wireless signals received from theexternal device to extract information therefrom. The transmit andreceive circuits 206, 208 are coupled to a controller 210 via respectiveelectrical connections 232, 234. In a transmit mode, the controller 210also provides information to the transmit circuit 206 for encoding andmodulating information into wireless signals. The transmit circuit 206communicates the wireless signals to the antenna 202 for transmission toan external device (e.g., server 110 of FIG. 1). In a receive mode, thereceive circuit 208 provides decoded wireless signal information to thecontroller 210. The controller 210 uses the decoded wireless signalinformation in accordance with the function(s) of the communicationdevice 200.

An antenna 240 is coupled to Global Navigation Satellite System (GNSS)device 214. The GNSS device 214 can include, but is not limited to, aGlobal Positioning System (GPS) receiver circuit for receiving GPSsignals. Those skilled in the art will appreciate that GPS is just oneform of a GNSS. Other types of GNSSs include GLONASS, Galileo, and/orBeiDou. The GNSS device 214 demodulates and decodes the signals toextract location information therefrom. The location informationindicates the location of the communication device 200. The GNSS device214 provides the decoded location information to the controller 210. Assuch, the GNSS device 214 is coupled to the controller 210 via anelectrical connection 236. The controller 210 uses the decoded locationinformation in accordance with the function(s) of the communicationdevice 200.

The controller 210 stores the decoded wireless signal information andthe decoded location information in a memory 212 of the communicationdevice 200. Accordingly, the memory 212 is connected to and accessibleby the controller 210 through an electrical connection 232. The memory212 may be a volatile memory and/or a non-volatile memory. For example,the memory 212 can include, but is not limited to, a Random AccessMemory (RAM), a Dynamic Random Access Memory (DRAM), a Static RandomAccess Memory (SRAM), a Read-Only Memory (ROM), and/or a flash memory.

As shown in FIG. 2, one or more sets of instructions 250 are stored inthe memory 212. The instructions 250 can also reside, completely or atleast partially, within the controller 210 during execution thereof bythe communication device 200. In this regard, the memory 212 and thecontroller 210 can constitute machine-readable media. The term“machine-readable media”, as used here, refers to a single medium ormultiple media that store the one or more sets of instructions 250. Theterm “machine-readable media”, as used here, also refers to any mediumthat is capable of storing, encoding or carrying the set of instructions250 for execution by the communication device 200 and that cause thecommunication device 200 to perform one or more of the methodologies ofthe present disclosure.

The controller 210 is also connected to a user interface 232. The userinterface 232 comprises input devices 216, output devices 224, andsoftware routines (not shown in FIG. 2) configured to allow a user tointeract with and control software applications 252 installed on thecommunication device 200. Such input and output devices respectivelyinclude, but are not limited to, a display 228, a speaker 226, a keypad220, a directional pad (not shown in FIG. 2), a directional knob (notshown in FIG. 2), a camera 218 and a microphone 222. The display 228 maybe designed to accept touch screen inputs.

The communication device 200 may further comprise a haptic feedbackelement 230, and/or a power source 260. The haptic feedback element 230can include, but is not limited to, a sound generator (e.g., a speaker),a visual alert generator (e.g., a light emitting diode(s)), a vibrationgenerator, and/or a haptic motor. All of the listed devices are wellknown in the art, and therefore will not be described here. The hapticfeedback element 230 is configured to provide users with auditory,visual and/or tactile notifications of what operations and/or functionshave been selected, and/or a status of certain operations and/orfunctions.

The power source 260 can include, but is not limited to, a battery, aninternal power generator, external power source, and/or an energyharvesting circuit. The energy harvesting circuit is generallyconfigured to harvest energy from a surrounding environment that can beused to power the electronic components of the communication device 200.The harvested energy can include, but is not limited to, light, RFenergy, vibration and/or heat.

Sensors 262 may also be provided with the communication device 200. Thesensors 262 can include, but are not limited to, cameras,accelerometers, vibration sensors, orientation sensors, temperaturesensors, humidity sensors, and/or odor/sent/smell sensors.

Referring now to FIG. 3, there is provided a detailed block diagram ofan illustrative architecture for the computing device 300. The MCD 106,server 110, wireless sensor node 114, data collection unit 118,computing device 116 and/or gateway 122 of FIG. 1 is/are the same as orsimilar to computing device 300. As such, the discussion of computingdevice 300 is sufficient for understanding devices 106, 110, 114, 116,118, 122 of FIG. 1.

Computing device 300 may include more or less components than thoseshown in FIG. 3. However, the components shown are sufficient todisclose an illustrative embodiment implementing the present solution.The hardware architecture of FIG. 3 represents one embodiment of arepresentative computing device configured to facilitate systemmanagement of railcar(s) and data collection unit(s). As such, thecomputing device 300 of FIG. 3 implements at least a portion of themethods described herein for associating a railcar to a data collectionunit.

Some or all the components of the computing device 300 can beimplemented as hardware, software and/or a combination of hardware andsoftware. The hardware includes, but is not limited to, one or moreelectronic circuits. The electronic circuits can include, but are notlimited to, passive components (e.g., resistors and capacitors) and/oractive components (e.g., amplifiers and/or microprocessors). The passiveand/or active components can be adapted to, arranged to and/orprogrammed to perform one or more of the methodologies, procedures, orfunctions described herein.

As shown in FIG. 3, the computing device 300 comprises a user interface302, a CPU 306, a system bus 310, a memory 312 connected to andaccessible by other portions of computing device 300 through system bus310, and hardware entities 314 connected to system bus 310. The userinterface can include input devices (e.g., a keypad 350) and outputdevices (e.g., speaker 352, a display 354, and/or light emitting diodes356), which facilitate user-software interactions for controllingoperations of the computing device 300.

At least some of the hardware entities 314 perform actions involvingaccess to and use of memory 312, which can be a RAM. Hardware entities314 can include a disk drive unit 316 comprising a computer-readablestorage medium 318 on which is stored one or more sets of instructions320 (e.g., software code) configured to implement one or more of themethodologies, procedures, or functions described herein. Theinstructions 320 can also reside, completely or at least partially,within the memory 312 and/or within the CPU 306 during execution thereofby the computing device 300. The memory 312 and the CPU 306 also canconstitute machine-readable media. The term “machine-readable media”, asused here, refers to a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store the one or more sets of instructions 320. The term“machine-readable media”, as used here, also refers to any medium thatis capable of storing, encoding or carrying a set of instructions 320for execution by the computing device 204 and that cause the computingdevice 300 to perform any one or more of the methodologies of thepresent disclosure.

In some scenarios, the hardware entities 314 include an electroniccircuit (e.g., a processor) programmed for facilitating the associationof a railcar to a data collection unit. In this regard, it should beunderstood that the electronic circuit can access and run a softwareapplication 322 installed on the computing device 300.

A wireless communication device 360 and/or a system interface 362 mayalso be provided with the computing device 300. The wirelesscommunication device 360 is configured to facilitate wirelesscommunications between the computing device 300 and external devices(e.g., remote server 110, MCD(s) 106, data collection unit 118, and/orgateway 122 of FIG. 1). The wireless communications can include, but arenot limited to, NFCs, SRCs (e.g., WiFi Bluetooth, and/or LoRA),satellite communications, and/or cellular communications. The systeminterface 362 is configured to facilitate wired and/or wirelesscommunications between the computing device 300 and external devices(e.g., remote server 110, MCD(s) 106, data collection unit 118, and/orgateway 122 of FIG. 1). In this regard, the system interface 362 caninclude, but is not limited to, an Ethernet interface, an RS232interface, an RS422 interface, and/or a USB interface.

Methods for Associating A Railway Asset to a Data Collection Unit

Referring now to FIG. 4A, there is provided a flow diagram of a method400 for associating a railway asset (e.g., a railcar) to a datacollection unit. Method 400 begins with 402 and continues with 404 wherean image is captured of a railway asset (e.g., railcar 102 of FIG. 1)using an MCD (e.g., MCD 106 of FIG. 1). The captured image is thenprocessed in 406 by the MCD to at least generate an image of a mark(e.g., a railcar mark) on a railway asset in an electronic editableform. The electronic editable mark can include one or more letters,number, graphics and/or symbols. The electronic editable mark can beedited and/or modified by a user of the MCD. In 408, the MCD receives auser input indicating whether or not the electronic editable markmatches a mark (e.g., mark 130 of FIG. 1) disposed on the railway assetby a certain amount (e.g., >75%). If the user input indicates that theelectronic editable mark does match the mark disposed on the railwayasset by the certain amount [410:NO], then method 400 continues with 412where the MCD receives a user input for modifying the electroniceditable mark. Otherwise [410:YES], method 400 continues with 414.

In 414, various information is sent from the MCD to a data collectionunit (e.g., data collection unit 118 of FIG. 1) via SRC(s) (e.g., anRFID communication and/or a Bluetooth communication). The presentsolution is not limited in this regard. Long Range Communications (LRCs)can be used in 414 as an alternative to or in addition to SRC(s). Thisinformation includes, but is not limited to, the mark information and/orother railway asset information. This information may also be sent to adata collection unit via a gateway and/or sensor. In 416, the markinformation, the railway asset information and/or metadata is sent fromthe data collection unit to a remote sever (e.g., remote sever 110 ofFIG. 1) via a device (e.g., gateway and/or CMU 122) and/or a network(e.g., network 108 of FIG. 1).

Next in 418, a determination is made as to whether the MCD hasconnectivity to the remote server. If not [418:NO], method 400 continuesto 432 of FIG. 4B, as shown by 420. The operations of 432 will bediscussed below.

If so [418:YES], then 422 is performed where the captured image, railwayasset information and/or metadata are communicated to the remote servervia the network. At the remote server, comparison operations may beperformed in 424 to compare the information received from the datacollection unit, the information received from the MCD, and/orpre-stored information to each other. The pre-stored information caninclude information stored in a datastore (e.g., datastore 112 ofFIG. 1) accessible to the remote server and/or a datastore that ishosted by a third party. In some scenarios, these operations can be atleast partially performed by the data collection unit.

If the comparison results indicate that a match does not exist betweencompared information [426:NO], method 400 continues with 430 where thereceived information is optionally discarded, the MCD is notified of avalidation failure, and/or the process returns to 402. The notificationmay provide a means to cause the MCD to prompt the individual (e.g.,individual 104 of FIG. 1) to move the MCD in a position such that therailway asset is in its camera's Field of View (FOV), detect when thecamera is in position, capture another image of the railway asset,generate another electronic editable mark for the railway asset, receivea user input regarding the accuracy and/or completeness of theelectronic editable mark, and/or communicate the electronic editablemark to the remote server (directly or indirectly).

In contrast, if the comparison results indicate that a match does existbetween compared information [426:YES], 428 is performed where avalidation is made that the data collection unit was installed on therailway asset. Thereafter, method 400 continues with 432 of FIG. 4B.

As shown in FIG. 4B, 432 involves storing the received information fromthe MCD in a datastore so that the mark of the railway asset and aunique identifier of the data collection unit are associated with eachother. Notably, in the scenarios where the MCD does not haveconnectivity to a network, the information is stored on the MCD untilconnectivity is restored, or until the data collection unit receives thedata and sends the same to the server. The received information may alsobe presented to a user of a computing device (e.g., sever 110 of FIG. 1and/or computing device 116 of FIG. 1), as shown by 434.

An electronic message may also be sent in 436 that notifies a computingdevice (e.g., computing device 116 of FIG. 1) of the installationverification. In response to the electronic message, the computingdevice may establish a communication session with the installed datacollection unit (e.g., data collection unit 118 of FIG. 1), remotelycause the data collection unit to perform a systems test and/orcalibration process, receive data resulting from performance of thesystems test and/or calibration process, identify or otherwise detectany system or operational faults/errors/issues based on the datareceived from the data collection unit, issue any alerts/notificationsto the individual (e.g., individual 104 of FIG. 1) regarding theoperational faults/errors/issues so that any remedial measures can betaken to address the system faults/errors/issues (e.g., re-initializesoftware, upgrade software, replace a part, or replace the entire datacollection unit).

Subsequently in 438, the data collection unit performs operations to,but not limited to, (i) monitor the performance and operation of therailway asset (e.g., a railcar), (ii) monitor statuses of components(e.g., hatches, valves, etc.) of the railway asset, (iii) monitor anamount of load disposed in/on the railway asset, and/or (iv) monitorconditions of an environment surrounding the railway asset. Suchoperations may be performed by the data collection unit in response to acommand received from the MCD, individual in possession of the MCD,and/or a remote device (e.g., computing device 116 of FIG. 1).Information is communicated in 440 from the data collection unit to theremote sever via a wireless sensor node (e.g., wireless sensor node 114of FIG. 1), a gateway (e.g., gateway 122 of FIG. 1) and/or a network(e.g., network 108 of FIG. 1). The information is analyzed in 442 todetermine a status of the railway asset, a status of at least onecomponent of the railway asset, and/or conditions of the environmentsurrounding the railway asset (e.g., temperature, humidity, amount oflight, presence of smoke, presence of fumes of given types, etc.).

If the status(es)/conditions is(are) acceptable [444:YES], then 446 isperformed where transportation activities for the railway asset arescheduled. For example, the status(es)/condition(s) is(are) consideredacceptable when all hatches/valves/doors are able to beopened/closed/locked/unlocked, the wheels/brakes/axles/railcar connector(e.g., a coupler) is(are) operating as expected, an amount of loaddisposed in the railway asset is within a given range, and/or the loadis being fully retained inside the railway asset (e.g., no leaks aredetected based on odors/scents/smells detected by an odor/scent/smellsensor).

In contrast, if the status(es)/condition(s) is(are) not acceptable[444:NO], then 448 is performed where the railway asset is removed fromuse temporarily, further image processing is performed to identifycomponent type(s) and/or component serial number(s), component(s)is(are) ordered based on the image processing, maintenance for therailway asset is scheduled and/or performed, and/or an amount of load ofthe railway asset is adjusted. Upon completing 444, 446 and/or 450 maybe performed. In 450, method 400 ends or other operations are performed(e.g., return to 402 of FIG. 4A).

The above described systems 100 and methods 400 can be implemented in ARbased railyard management systems and/or other AR based systems. Onesuch AR based railyard management systems is discussed below in whichthe present solution can be implemented.

Systems for AR Based Railyard Management

Referring now to FIG. 5, there is provided an illustration of anothersystem 500. System 500 can include some or all of the components ofsystem 100 of FIG. 1. The relationship between systems 100 and 500 willbecome evident as the discussion progresses.

As shown in FIG. 5, system 500 comprises a railyard 502 with one or moretracks 504. Railway asset(s) 506 may reside on track(s) 504. Anillustration is provided in FIG. 6 that shows a set of tracks. Track(s)504 can include, but are not limited to, those shown in FIG. 6. Morespecifically, the tracks of the railyard 502 can include main tracks600, 632, inbound tracks 602-610, and outbound tracks 622-630.Load/unload areas 612-620 can be designated on the inbound tracks602-610 and/or outbound tracks 622-630.

During operation, a railway asset 506 enters the railyard 502 via maintrack 600. In some scenarios, the railway asset comprises a railcar 542that may be part of a train consist 540. The train consist 540 comprisesa plurality of railcars coupled to each other. The train consist 540 caninclude, but is not limited to, the locomotive 120. The train consist isdecoupled and disassembled on the main track 600 into individualrailcars 542. Railcars 542 can include, but are not limited to, railcar102 of FIG. 1. The individual railcars are then moved to one or more ofthe inbound tracks 602-610, for example, based on their classificationsand/or train consist reassembly sequences.

While on the track(s), individual(s) 514 inspect the railcars 542 usingvirtual reality device(s) 508. Individual(s) 514 can include, butis(are) not limited to, individual(s) 104 of FIG. 1. The virtual realitydevice(s) 508 can comprise MCD 106 of FIG. 1. The virtual realitydevice(s) 508 may include, but is(are) not limited to, virtual realityheadset(s) (e.g., Microsoft Hololens), smart glasses implementing amixed reality platform, and/or other devices implementing mixed realityplatforms (e.g., an electric transportation device (e.g., a Segway®)with a transparent screen in the form of a windshield).

Virtual reality headsets and smart glasses are well known. It should beunderstood that the virtual reality headsets and smart glasses includetransparent screens in the form of lenses for an augmented realityexperience. The individual(s) 514 is(are) able to experience 2D and/or3D holographic image(s) as though they are part of the real worldrailyard environment. The 2D and/or 3D holographic image data isintegrated with real world data, and displayed to the individual(s) viaone or more transparent screens. The 2D and/or 3D holographic image datacan include, but is not limited to, railyard map information, railcarinformation, life-cycle analysis information, inspection information,maintenance scheduling information, requirement compliance information,and/or security check information.

In some scenarios, the virtual reality device(s) 508 is(are) used toautomate (i) railyard map updates with railcar locations by track andrailcar status, (ii) maintenance inspections, and/or (iii) maintenancescheduling. The automated inspections can include, but are not limitedto, wheelset inspections, hand brake inspections, piston pin travelinspections, spring nest inspections, bearing inspections, and/orrailcar body appliance inspections. For example, the virtual realitydevice(s) 508 is(are) used to: (i) detect whether or not handbrakes ofthe railcar(s) are set before loading/unloading operations are startedin the load/unload areas 612-620; (ii) detect whether or not tank carswheels are chocked before loading/unloading operations are started inthe load/unload areas 612-620; (iii) detect any damage to the railcar(e.g., a crack in a wheelset, a car body, etc.); and/or (iv) detectwhether or not a valve or manway is closed and locked after completionof loading/unloading operations. The present solution is not limited tothe particulars of this example.

The virtual reality device(s) 508 may also be used to verify governmentrequirements, shipping requirements, and/or customer requirements aremet. For example, the virtual reality device(s) 508 may be used toverify that tank capacities match loading tickets, verify productquantities, verify product qualities, ensure place cards andidentification numbers are legible and correct, verify special permitnumbers are marked on railway assets, verify that proper shipping namesare marked on railway assets, verify that inhalation hazards are markedon railway assets, and/or verify certain decals are present on railwayassets.

The virtual reality device(s) 508 may further be used to automatesecurity checks of the containers and/or report any unusual conditions.The containers can include, but are not limited to, shipping containers,ISO containers, and/or any other freight transport item that forms partof and/or can be loaded onto a railcar or other railway asset. Theunusual conditions can include, but are not limited to, missing orbroken security seals, defective ladders, defective handles, defectivehandrails, defective tank shells, defective jacket heads, defectivedouble shelf couplers, defective Automatic Equipment Identification(AEI) tags, defective axles, defective wheels, and/or the presence ofsuspicious packages on or adjacent to the railway assets.

The information collected by the virtual reality device(s) 508 can beused by a machine learning algorithm to make predictions of futureevents relating to the railway asset. For example, the machine learningalgorithm can process information specifying current conditions ofcomponents of a railcar 542 to detect patterns which have been machinelearned to lead to a particular event with a certain degree oflikelihood or probability (e.g., a hairline crack in a wheel can lead toa derailment of the railcar with a certain degree of likelihood orprobability that exceeds a threshold value, or an offset in a bracketposition relative to a given reference point can lead to a mechanicalfailure of an axle with a certain degree of likelihood or probabilitythat exceeds a threshold value). The present solution is not limited tothe particulars of this example. The machine learning algorithm can beperformed by one or more of the devices 508, 510, 528, 530 of FIG. 5.

The machine-learning algorithm(s) can employ supervised machinelearning, semi-supervised machine learning, unsupervised machinelearning, and/or reinforcement machine learning. Each of these listedtypes of machine-learning algorithms is well known in the art. In somescenarios, the machine-learning algorithm includes, but is not limitedto, a decision tree learning algorithm, an association rule learningalgorithm, an artificial neural network learning algorithm, a deeplearning algorithm, an inductive logic programming based algorithm, asupport vector machine based algorithm, a Bayesian network basedalgorithm, a representation learning algorithm, a similarity and metriclearning algorithm, a sparse dictionary learning algorithm, a geneticalgorithm, a rule-based machine-learning algorithm, and/or a learningclassifier system based algorithm. The machine-learning processimplemented by the present solution can be built usingCommercial-Off-The-Shelf (COTS) tools (e.g., SAS available from SASInstitute Inc. of Cary, N.C.).

The virtual reality device(s) 508 are configured to communicate withexternal device(s) 510 via wireless communications for facilitating theautomation of the above described processes/tasks. For example, thevirtual reality device(s) 508 is(are) configured to communicate withcommunication device(s) 510 via wireless communication links 512,516/520, 522/526. The communication device(s) 510 can include, butis(are) not limited to, gateways, mobile devices (e.g., radios, tablets,smart phones, etc.), and/or other devices. The wireless communicationscan include, but are not limited to, satellite communications, LRCs(e.g., cellular communications and/or WiFi communications) and/or SRCs(e.g., Bluetooth).

The virtual reality device(s) 508 is(are) also configured to communicatewith remote server(s) 528 via network 524 (e.g., an Intranet orInternet). Remote server can include, but is not limited to, server 110of FIG. 1. Network 524 can include, but is not limited to, network 108of FIG. 1. The remote server(s) 528 is(are) configured to (i) facilitateaccess to and storage of data 534 in datastore 532 (e.g., a database),and/or (ii) facilitate the provision of notifications and/or alerts tocomputing device(s) 530 (e.g., of site supervisors or managers).Datastore 532 can include, but is not limited to, datastore 112 ofFIG. 1. The notifications and/or alerts can concern detected defects ofrailway assets, maintenance scheduling for railway assets, locations ofrailway assets, statuses of railway assets, detected unusual activity inrailyards, inspection statuses, inspection results, security checkstatuses, security check results, and/or requirementsatisfaction/compliance. The remote server(s) 528 can use thisinformation to update railyard maps and/or railyard GUIs in real time ornear real time (e.g., as the railyard inspection(s) is(are) beingperformed). Illustrative GUIs are shown in FIGS. 10-12 and 15-18.

In some scenarios, the information acquired by the virtual realitydevice(s) 508 can also be used to facilitate the inspection andinteraction with real-world railway assets in real-time or nearreal-time using virtual reality technology by individual(s) 556 (e.g.,mechanics) located at a site 550 remote from the railyard 502. Forexample, a computing device 530 obtains a digital 3D model of thereal-world railyard environment including railway asset(s) and/orrobotic manipulator(s) 536 from the datastore 532 via server 528.Robotic manipulator(s) are well known (e.g., an articulating ortelescoping arm with a gripper at a free end). A video is generated by acamera of the virtual reality device(s) 508 and/or other camera 538placed in the railyard 502, and streamed to the computing device 530.The computing device 530 uses the video's content to convert the digital3D model into another digital 3D model representative of the currentlocations, positions and/or orientations of the real-world railwayasset(s) and robotic manipulator(s). The individual 556 is then providedwith a real-time or near real-time virtual reality experience with thereal-world railway asset(s) and robotic manipulator(s) by displaying thedigital 3D model in a virtual reality environment 554. The individual556 can cause movement of the robotic manipulator(s) in the railyard 502via user-software interactions for interacting the with digital 3D modelwhile the individual is having the real-time or near real-time virtualreality experience at site 550. In this way, maintenance of a railwayasset can be achieved through the remote control of the roboticmanipulator(s) via virtual reality technology.

Once the inspection and safety check is completed, the railcar 542 ismoved to an outbound track (e.g., outbound track 622 of FIG. 6). Therailcar 542 is then moved to the main track 632 in accordance with atrain consist reassembly sequence. The train consist may then beassembled, verified by the virtual reality device(s) 508, and leave therailyard 502 via main track 632.

Virtual reality device(s) 508, communication device(s) 510, server(s)528 and/or computing device(s) 530 of FIG. 5 is(are) the same as orsubstantially similar to communication device 200 of FIG. 2 and/orcomputing device 300 of FIG. 3. The above discussion of device(s) 200,300 is(are) sufficient for understanding device(s) 508, 510, 528, 530 ofFIG. 5. Still, it should be noted that display 354 can include a VRdisplay apparatus, and the hardware entities 214 can include anelectronic circuit (e.g., a processor) programmed for facilitating theprovision of AR based railyard management and/or a VR environment inwhich a visual experience with the real-world railyard assets/structurescan be simulated in real-time or near real-time. The electronic circuitcan access and run software application(s) that is(are) installed on thedevice 200, 300 and generally operative to facilitate automation of thefollowing tasks, but not limited to: validating train consists,validating information and/or markings on railway assets,detecting/tracking locations of railway assets in railyards, updatingrailyard maps, monitoring states of the railway assets while in therailyards, detecting damage to railway assets in the railyards,detecting hazards of railway assets, predicting future issues withrailway assets based on machine learned information (e.g., a predictedderailment of a railcar based on detected state(s) of components thereof(e.g., a detected crack or other mechanical fault in a wheel, axle,bearing, etc.)), performing maintenance checks for various components ofthe railway assets (e.g., wheels, axles, bearings, appliances, ladders,etc.), scheduling maintenance for the railway assets, facilitatingmaintenance of railway assets using robotic manipulator(s) (e.g.,articulating arms) which are remotely controlled via a VR environment,performing security checks, performing compliance checks (e.g.,regulatory, shipping, customer, etc.), and providingalerts/notifications to relevant parties/individuals. Other functions ofthe these software application(s) are apparent from the presentdisclosure and drawings. Such other functions can relate to remotecontrol of moving parts in a railyard (e.g., robotic manipulator(s) 536of FIG. 5) and/or operational parameters. The software application(s)is(are) operative to access parameter(s)/requirement(s) stored in memoryof the device 200, 300 and/or data (e.g., data 534 of FIG. 5) stored ina remote datastore (e.g., datastore 532 of FIG. 5).

FIGS. 7-8 provide illustrations that are useful for understanding thetypes and locations of information, markings and decals printed orotherwise disposed on a railway asset (e.g., railcar 102 of FIG. 5and/or railcar 542 of FIG. 5). The types and locations of information,markings and decals that can be used in conjunction with the presentsolution are not limited to that shown in FIGS. 7-8. Also, theinformation included in FIGS. 7-8 may change in accordance with industrystandards and railway asset type.

Markings specifying the following information may be printed orotherwise disposed on the railway asset at respective locations: aleasing company identifier; a railway asset number, an authorizingagency identifier (e.g., Department of Transportation (DOT), Associationof American Railroads (AAR), and/or Transport Canada (TC)); a classdesignation (e.g., non-pressure tank cars, cryogenic liquid tank cars,pressure tank car, multi-unit tank car (containers), high pressure tankcar, pneumatically unloaded covered hoppers, and/or wooden tank car);separator character (e.g., top and bottom shelf couplers, tankheadshields, jacketed thermal protection, and/or spray-on thermalprotection); tank test pressure; material type used in tank construction(e.g., carbon steel, aluminum, aluminum alloy, nickel, and/or stainlesssteel alloy); type of weld (e.g., fusion weld or forge weld); and/orother car features (e.g., fittings, materials, linings).

The AR/VR application(s) (e.g., applications 322 of FIG. 3) areoperative to recognize such markings on railway assets based on imagesand/or videos captured by VR device(s) (e.g., MCD 106 of FIG. and/or VRdevice(s) 508 of FIG. 5), and perform automated railyard managementoperations/tasks using content of the recognized markings. For example,the AR/VR application(s) may cause a railyard map to be updated toinclude an icon representing a given railcar and showing a currentlocation of a railcar in the railyard. Content of the recognizedmarkings and/or collected railcar status information can be superimposedon the railyard map or otherwise displayed on a screen in response to auser-software action for selecting the icon. The present solution is notlimited in this regard.

An illustrative railyard map 900 is shown in FIG. 9. The railyard map900 comprises railcar icons 902 arranged to show current locations ofrailcars on tracks of the railyard and to show relative positions (orsequenced order) of railcars on each track. An illustrative window 1000showing content of recognized markings (e.g., railcar mark, LD limit,and tare weight), railcar location/position information (e.g., track 1,position 1), and collected railcar status information (e.g.,passed/failed inspection, was/was not unloaded/loaded) is provided inFIG. 10.

Referring now to FIG. 19, there is provided a flow diagram of anillustrative method 1900 for AR based railyard management. Method 1900begins with 1902 and continues with 1904 where an individual (e.g.,individual 104 of FIG. 1 and/or 514 of FIG. 5) physically inspects areal world railway asset (e.g., train consist 504 of FIG. 5 and/orrailcar 542 of FIG. 5). During the inspection, the real world railyardenvironment is visible to the individual via a transparent screen of avirtual reality device (e.g., MCD 106 of FIG. 1 and/or virtual realitydevice 508 of FIG. 5), as shown by 1906.

In 1908, the virtual reality device performs operations to obtain realworld information about the railway asset and/or railyard (e.g.,railyard 502 of FIG. 5) in real time. This information can be obtainedusing a camera (e.g., camera 218 of FIG. 2) and/or other sensor(s)(e.g., sensor(s) 262 of FIG. 2) of the virtual reality device.Additionally or alternatively, the information can be obtained viawireless communications between the virtual reality device and a datacollection unit 118 installed or being installed on the railway asset.The information can include, but is not limited to, information disposedon the railway asset(s) (e.g., railcar mark(s)), tare weight(s), maximumweight(s), certification reference number(s), certification date(s),data collected from sensors installed on the railcar (e.g., commoditytemperature sensor data, hatch status data, bearing temperature data,and/or load status data), railcar/locomotive component data (e.g., make,model, serial number, wheel size, etc.), maintenance information (e.g.,date/time of last maintenance and/or type of maintenance performed),location of railway asset, and/or information indicating state(s) and/orcondition(s) of the railway asset and/or component(s) thereof. Thestate(s)/condition(s) can include, but are not limited to, satisfactorymechanical state(s)/condition(s) (e.g., the railway asset has properlyoperating brakes, hatches, discharge gates, ports, doors, valves, ports,manways, etc.), mechanical fault state(s)/condition(s) (e.g., defectivebrakes, hatches, discharge gates, doors, valves, ports and/or manways topercent proper operation), satisfactory physical state(s)/condition(s)(e.g., absent of dents, cracks, holes, wear, etc.), damagedstate(s)/condition(s) (e.g., presence of dents, cracks, holes, wear,etc.), safe state(s)/condition(s) (e.g., hatches, discharge gates,ports, doors, valves, ports, manways, etc. are properly sealed, closedand/or latched), unsafe state(s)/condition(s) (e.g., hatches, dischargegates, ports, doors, valves, ports, manways, etc. are not sealed, areopen and/or are unlatched), and/or hazardous state(s)/condition(s)(e.g., a particular type and/or amount of smell/scent detected outsideof railway asset, and/or a tripping or fall hazard exists, for example,due to a broken or missing rung on a ladder).

In optional 1910, the real world information is used to, but not limitedto, update a railway map (e.g., railyard map 900 of FIG. 9), railwayasset information (e.g., information shown in FIGS. 7-8), life cyclecompliance information, inspection information, maintenance schedulinginformation, requirement compliance information (e.g., government and/orcustomer), and/or security check information. This operation can beperformed by the virtual reality device and/or another device (e.g.,server 110 of FIG. 1, computing device 116 of FIG. 1, communicationdevice 510 of FIG. 5, server 528 of FIG. 5, and/or communication device530 of FIG. 5) that is communicatively coupled to the virtual realitydevice. The updated information can be stored in a datastore (e.g.,datastore 112 of FIG. 1, memory 212 of FIG. 2, memory 312 of FIG. 3and/or datastore 532 of FIG. 5).

In 1912, holographic image data is generated using the updatedinformation. The holographic image data is then displayed on thetransparent screen of the virtual reality device so that the individualis provided with a holographic AR experience, as shown by 1914. Theoperations of 1908-1914 may be repeated a number of times while therailway asset is being inspected, as shown by 1916.

The real world information may optionally be used in 1918 to predictfuture event(s) and/or condition(s) relating to the railway asset. Theprediction can be made using machine learning algorithms. The machinelearning algorithms can, for example, be trained to predict derailmentof a railcar based on detected state(s) of components thereof (e.g., adetected crack or other mechanical fault in a wheel, axle, etc.), and/orpredict component failure (e.g., based on life expectancy thereof,component type, duration of use, and/or amount of wear/tear from use).The predicted future event(s) and/or condition(s) may optionally beoutput from the virtual reality device and/or from another device, asshown by 1920. In 1922, the predicted future event(s) and/orcondition(s) may be used to cause action(s) and/or task(s) to be takenwhich relate to the railway asset. For example, a part for the railwayasset can be ordered and/or maintenance of the railway asset can bescheduled. Additionally or alternatively, an adjustment to the amount ofload in/on the railyard asset is caused and/or a temporary removal ofthe railway asset from use is caused. The present solution is notlimited to the particulars of this example.

In 1924, the real world information may optionally be used to facilitatean inspection and/or interaction with the real world railway asset inreal-time or near real-time using virtual reality technology by anotherindividual (e.g., individual 556 of FIG. 5) located at a site (e.g.,site 550 of FIG. 5) remote from the railyard.

For example, a computing device at the remote site obtains a digital 3Dmodel of the real-world railyard environment including railway asset(s)and/or robotic manipulator(s) from the datastore via a server. A videois generated by a camera of the virtual reality device and/or othercamera placed in the railyard, and streamed to the computing device. Thecomputing device uses the video's content to convert the digital 3Dmodel into another digital 3D model representative of the currentlocations, positions and/or orientations of the real-world railway assetand a robotic manipulator (e.g., robotic manipulator 536 of FIG. 5). Theindividual is then provided with a real-time virtual reality experiencewith the real-world railway asset and robotic manipulator by displayingthe digital 3D model in a virtual reality environment. The individualcan cause movement of the robotic manipulator in the railyard viauser-software interactions for interacting the with digital 3D modelwhile the individual is having the real-time virtual reality experienceat the remote site. In this way, maintenance of a railway asset can beachieved through the remote control of the robotic manipulator(s) viavirtual reality technology. Subsequently, 1926 is performed where method1900 ends or other operations are performed (e.g., return to 1902).

All of the apparatus, methods and algorithms disclosed and claimedherein can be made and executed without undue experimentation in lightof the present disclosure. While the invention has been described interms of preferred embodiments, it will be apparent to those of skill inthe art that variations may be applied to the apparatus, methods andsequence of steps of the method without departing from the concept,spirit and scope of the invention. More specifically, it will beapparent that certain components may be added to, combined with, orsubstituted for the components described herein while the same orsimilar results would be achieved. All such similar substitutes andmodifications apparent to those skilled in the art are deemed to bewithin the spirit, scope and concept of the invention as defined.

We claim:
 1. A method for railway asset management, comprising:capturing an image of the railway asset using a mobile communicationdevice; converting the image into an electronic editable image for therailway asset; communicating the electronic editable image from themobile communication device to a data collection unit which is installedon the railway asset; communicating first information from the datacollection unit to a remote computing device via a first networkcommunication, the first information comprising at least the electroniceditable image; comparing the first information to second information todetermine whether a match exists therebetween by a given amount; andvalidating that the data collection unit was installed on the railwayasset when a match is determined to exist between the first and secondinformation by the given amount.
 2. The method according to claim 1,further comprising communicating the second information from the mobilecommunication device to the remote computing device with a secondnetwork communication, the second information comprising at least theimage.
 3. The method according to claim 1, wherein the secondinformation is pre-stored information retrieved from a datastore of arailway asset management system or a datastore of another system.
 4. Themethod according to claim 1, further comprising providing an electronicnotification to a user of a computing device that the install wascompleted successfully, when a match is determined to exist between thefirst and second information by the given amount.
 5. The methodaccording to claim 1, further comprising providing an electronicnotification to a user of a computing device that the install was notcompleted successfully, when a match is determined to not exist betweenthe first and second information by the given amount.
 6. The methodaccording to claim 1, further comprising storing the first informationin a datastore responsive to a validation that the data collection unitwas installed on the railway asset.
 7. The method according to claim 1,further comprising associating a unique identifier of the datacollection unit with a mark represented in the electronic editable imagein a datastore responsive to a validation that the data collection unitwas installed on the railway asset.
 8. The method according to claim 7,further comprising discarding the first information when a determinationis made that the first and second information do not match each other bythe given amount.
 9. The method according to claim 1, further comprisingperforming monitoring operations by the data collection unit in responseto said validating to monitor at least one of an operational performanceof the railway asset, a status of at least one component of the railwayasset, an operation of at least one component of the railway asset, anamount of load disposed in or on the railway asset, and a condition ofan environment surrounding the railway asset.
 10. The method accordingto claim 9, further comprising analyzing information from saidmonitoring operations to determine whether at least one of theoperational performance of the railway asset, the status of the at leastone component of the railway asset, and the condition of the environmentsurrounding the railway asset is acceptable.
 11. The method according toclaim 10, wherein the operational performance of the railway asset, thestatus of the at least one component of the railway asset, or thecondition of the environment surrounding the railway asset is consideredacceptable when at least one of (i) a hatch, a valve, a door, a wheel, abrake, an axle or a railway asset connection is operating in an expectedmanner, (ii) an amount of load disposed in or on the railway asset iswithin a given range, and (iii) no leaks have been detected based onodors, scents or smells detected by a sensor of the data collectionunit.
 12. The method according to claim 10, further comprisingperforming at least one of the following operations when a determinationis made that at least one of the operational performance of the railwayasset, the status of the at least one component of the railway asset,and the condition of the environment surrounding the railway asset isunacceptable: remove the railcar from use temporarily; order a componentbased on an analysis of the image; schedule maintenance for the railwayasset; and adjust an amount of load in or on the railway asset.
 13. Themethod according to claim 10, further comprising schedulingtransportation activities for the railway asset when a determination ismade that at least one of the operational performance of the railwayasset, the status of the at least one component of the railway asset,and the condition of the environment surrounding the railway asset isacceptable.
 14. A system, comprising: a data collection unit installableon a railway asset and configured to receive an electronic editableimage for a railway asset from an external device; and a computingdevice located remote from the data collection unit and configured to:receive first information communicated from the data collection unit viaa first network communication, the first information comprising at leastthe electronic editable image; compare the first information to secondinformation to determine whether a match exists therebetween by a givenamount; and validate that the data collection unit was installed on therailway asset when a match is determined to exist between the first andsecond information by the given amount.
 15. The system according toclaim 14, wherein the computing device is further configured to receivethe second information communicated from the external device via asecond network communication, the second information comprising at leastan image of the railway asset that was captured by the external device.16. The system according to claim 14, wherein the second information ispre-stored information retrieved from a datastore of a railway assetmanagement system or a datastore of another system.
 17. The systemaccording to claim 14, wherein the computing device is furtherconfigured to provide an electronic notification to a user of anothercomputing device that an install was completed successfully, when amatch is determined to exist between the first and second information bythe given amount.
 18. The system according to claim 14, wherein thecomputing device is further configured to provide an electronicnotification to a user of another computing device that an install wasnot completed successfully, when a match is determined to not existbetween the first and second information by the given amount.
 19. Thesystem according to claim 14, wherein the computing device is furtherconfigured to store the first information in a datastore responsive to avalidation that the data collection unit was installed on the railwayasset.
 20. The system according to claim 14, wherein a unique identifierof the data collection unit is associated with a mark represented in theelectronic editable image in a datastore responsive to a validation thatthe data collection unit was installed on the railway asset.
 21. Thesystem according to claim 20, wherein the first information is discardedwhen a determination is made that the first and second information donot match each other by the given amount.
 22. The system according toclaim 14, wherein the data collection unit is further configured toperform monitoring operations in response to said validating to monitorat least one of an operational performance of the railway asset, astatus of at least one component of the railway asset, an operation ofat least one component of the railway asset, an amount of load disposedin or on the railway asset, and a condition of an environmentsurrounding the railway asset.
 23. The system according to claim 22,wherein information from said monitoring operations is analyzed todetermine whether at least one of the operational performance of therailway asset, the status of the at least one component of the railwayasset, and the condition of the environment surrounding the railwayasset is acceptable.
 24. The system according to claim 23, wherein theoperational performance of the railway asset, the status of the at leastone component of the railway asset, or the condition of the environmentsurrounding the railway asset is considered acceptable when at least oneof (i) a hatch, a valve, a door, a wheel, a brake, an axle or a railcarconnection is operating in an expected manner, (ii) an amount of loaddisposed in or on the railway asset is within a given range, and (iii)no leaks have been detected based on odors, scents or smells detected bya sensor of the data collection unit.
 25. The system according to claim24, wherein at least one of the following operations is performed by thecomputing device when a determination is made that at least one of theoperational performance of the railway asset, the status of the at leastone component of the railway asset, and the condition of the environmentsurrounding the railway asset is unacceptable: cause the railway assetto be removed from use temporarily; cause a component to be orderedbased on an analysis of the image; cause maintenance for the railwayasset to be scheduled; and cause an amount of load in or on the railwayasset to be adjusted.
 26. The system according to claim 24, wherein thecomputing device causes transportation activities for the railway assetto be scheduled when a determination is made that at least one of theoperational performance of the railway asset, the status of the at leastone component of the railway asset, and the condition of the environmentsurrounding the railway asset is acceptable.
 27. A method for AugmentedReality (AR) based railyard management, comprising: using a virtualreality device to recognize and collect real world information aboutrailway assets located in a railyard; and using the real worldinformation to provide an individual with an augmented realityexperience associated with the railyard and facilitate automatedrailyard management tasks.
 28. The method according to claim 27, whereinthe railway asset comprises a train consist, a railcar, a locomotive, arail maintenance equipment, a container, or an International StandardsOrganization (ISO) tank.
 29. The method according to claim 27, whereinthe automated railyard management tasks comprises at least one ofvalidating a train consist, validating information disposed on therailway assets, detecting locations of the railway assets in therailyard, updating a map of the railyard, monitoring states of therailway assets while in the railyards, detecting damage to the railwayassets in the railyards, detecting hazards of the railway assets,predicting future issues with the railway assets based on machinelearned information, performing maintenance checks for components of therailway assets, scheduling maintenance for the railway assets,facilitating maintenance of railway assets using a robotic manipulatorwhich is remotely controlled via a virtual reality environment,performing security checks for the railyard, performing security checksfor the railway assets, performing compliance checks for the railwayassets, and providing notifications to individuals.
 30. The methodaccording to claim 27, wherein the augmented reality experience isprovided to the individual by allowing a real world environment of therailyard to be visible to an individual who is wearing the virtualreality device.
 31. The method according to claim 30, wherein theaugmented reality experience is provided to the individual by furthergenerating holographic image data using the real world information. 32.The method according to claim 31, wherein the real world informationcomprises at least one of locations of the railway assets, informationdisposed on the railway assets, physical conditions of the railwayassets, operating states of components of the railway assets, andphysical conditions of the components of the railway assets.
 33. Themethod according to claim 31, wherein the augmented reality experienceis provided to the individual by further overlaying the holographicimage data on the visible real world environment.
 34. The methodaccording to claim 27, further comprising performing a machine learningalgorithm using the real world information to predict a future event orcondition relating to at least one railway asset of the railway assets.35. The method according to claim 34, further comprising causing anaction to be taken in relation to the railway asset based on thepredicted future event or condition.
 36. The method according to claim27, further comprising using the real world information to facilitate aninspection of the railway assets by an individual remote from therailyard.
 37. The method according to claim 27, further comprising usingreal world information to facilitate a remote control of a roboticmanipulator located in the railyard.
 38. A system, comprising: aprocessor; a non-transitory computer-readable storage medium comprisingprogramming instructions that are configured to cause the processor toimplement a method for Augmented Reality (AR) based railyard management,wherein the programming instructions comprise instructions to: recognizeand collect real world information about railway assets located in arailyard; use the real world information to provide an individual withan augmented reality experience associated with the railyard andfacilitate automated railyard management tasks.
 39. The system accordingto claim 38, wherein the railway asset comprises a train consist, arailcar, a locomotive, a rail maintenance equipment, a container, or anInternational Standards Organization (ISO) tank.
 40. The systemaccording to claim 38, wherein the automated railyard management taskscomprises at least one of validating a train consist, validatinginformation disposed on the railway assets, detecting locations of therailway assets in the railyard, updating a map of the railyard,monitoring states of the railway assets while in the railyards,detecting damage to the railway assets in the railyards, detectinghazards of the railway assets, predicting future issues with the railwayassets based on machine learned information, performing maintenancechecks for components of the railway assets, scheduling maintenance forthe railway assets, facilitating maintenance of railway assets using arobotic manipulator which is remotely controlled via a virtual realityenvironment, performing security checks for the railyard, performingsecurity checks for the railway assets, performing compliance checks forthe railway assets, and providing notifications to individuals.
 41. Thesystem according to claim 38, wherein the augmented reality experienceis provided to the individual by allowing a real world environment ofthe railyard to be visible to an individual who is wearing a virtualreality device.
 42. The system according to claim 41, wherein theaugmented reality experience is provided to the individual by furthergenerating holographic image data using the real world information. 43.The system according to claim 42, wherein the real world informationcomprises at least one of locations of the railway assets, informationdisposed on the railway assets, physical conditions of the railwayassets, operating states of components of the railway assets, andphysical conditions of the components of the railway assets.
 44. Thesystem according to claim 42, wherein the augmented reality experienceis provided to the individual by further overlaying the holographicimage data on the visible real world environment.
 45. The systemaccording to claim 38, wherein the programming instructions furthercomprise instructions to perform a machine learning algorithm using thereal world information to predict a future event or condition relatingto at least one railway asset of the railway assets.
 46. The systemaccording to claim 45, wherein the programming instructions furthercomprise instructions to cause an action to be taken in relation to therailway asset based on the predicted future event or condition.
 47. Thesystem according to claim 38, wherein the programming instructionsfurther comprise instructions to use the real world information tofacilitate an inspection of the railway assets by an individual remotefrom the railyard.
 48. The system according to claim 38, wherein theprogramming instructions further comprise instructions to use real worldinformation to facilitate a remote control of a robotic manipulatorlocated in the railyard.